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1.0 Executive Summary 

In June 2004, the June Space Flight Leadership Council (SFLC) assigned an action to the NASA 
Engineering and Safety Center (NESC) and External Tank (ET) project jointly to characterize 
the available dataset [of defect sizes from dissections of foam], identify resultant limitations to 
statistical treatment of ET as-built foam as part of the overall thermal protection system (TPS) 
certification, and report to the Program Requirements Change Board (PRCB) and SFLC in 
September 2004. The NESC statistics team was formed to assist the ET statistics group in 
August 2004. 

Based on previous flight history and early destructive analysis (dissections), decisions were 
made to remove manually sprayed foam from several closeout areas. These “remove and 
replace” areas have foam reapplied using enhanced processes and more rigorous controls. 

Approximately 95 percent of the foam is being left on the tanks, as it was originally sprayed, 
with no rework or repair. These areas consist mainly of “acreage foam” applied with automated 
processes, but a large number of manually sprayed areas will also be left “as is”. 

In its simplest terms, the certification approach taken by the ET project for the debris 
requirement (cohesive failure mode) is to detennine, by testing, those combinations of defect 
size, shape, and depth within the foam which can cause a divot (foam debris). 

The joint statistical team defined the data available, specified the assumptions that must be made 
to establish statistical descriptions of the dissection data, and defined limitations on using 
statistics for certification. Choosing a distribution to represent the observed dissection data is a 
critical step in the statistical analysis. There is currently no engineering rationale to choose one 
distribution over another. Several distributions may fit the data equally well, and provide very 
different estimates of the expected maximum defect size. 

In addition, the NESC technical consultation team concluded that: 

1 . The largest characteristic defect size (CDS) value in the next sample is the quantity of interest, 
and should be estimated from the distribution of maximum values. 

2. Using the distribution of sample values in place of the distribution of the maximum values of a 
sample can raise the error probability by orders of magnitude. 

3. The critical information needed to predict maximum CDS for the next ET is the shape of the 
upper tail of the distribution of defect sizes. 
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Adoption of a standard distribution cannot be justified, so this shape is unknown and cannot be 
effectively constrained by the existing dissection data. Based on these findings, the technical 
consultation team found that there is no firm basis for using statistical analysis of the dissected 
data from ET 94 to serve as a statistical distribution model for the certification of “as-built foam” 
for other tanks. 


2.0 Identification 


A/I/C #: 04-027-1 

Initiator Name: 

Initiator Contact Info: 

Randy Galloway 

Thomas . R. Gallowav(fi)nasa. gov 
228-332-3281 

Short Title: Technical Consultation of the Statistical Analysis Support Provided for the 
External Tank (ET) Project. Note: After the NRB presentation on May 26, 2005, it was 
recommended to change the title to Support Provided to the External Tank (ET) Project on 
the Use of Statistical Analysis for ET Certification 

Description: The June 2004 Space Flight Leadership Council (SFLC) assigned an action to 
the NASA Engineering and Safety Center (NESC) and the External Tank (ET) project to 
characterize the available dataset, identify resultant limitations to statistical treatment of ET 
as-built foam as part of the overall Thermal Protection System (TPS) certification, and report 

to the PRCB and SFLC. 


Date Received: June 2004 

Date A/I/C Initiated: June 2004 

Initial Evaluators Assigned: 

Initial Evaluators Contact Info: 

NA 

NA 

Lead Assigned: 

Lead Contact Info: 

Cynthia H. Null 

Cynthia.H.Null@nasa.gov 

650-604-1260 
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2.1 Team Members, Consultants, and Advisors 


Last 

First 

Position/SPRT 

Center/ 

Office 


Name 

Name 

Affiliation 

Contractor 

Number 

Email 
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Cynthia 
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650-604-1260 

Cynthia.H.Null@nasa.gov 

Conway 
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Consultant, Statistical 
Evaluation of Shuttle 
External Tank 
Processing 

Swales 

Aerospace 
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baconway96@aol.com 

Scargle 

Jeffrey 

Space Science 
Planetary Branch, 
Statistical Data 
Analysis 

ARC 

650-604-6330 

Jeffrey.D.Scargle@nasa.gov 

McAlhaney 

Lisa 

Management & 
Technical Support 
Office (MTSO) 

LaRC 

757-864-2139 

Lisa.A.McAlhaney@nasa.gov 

Advisors: 

Ahumada 

A1 

Human Factors SPRT 

ARC 

650-604-6257 

Al.Ahumada@nasa.gov 


3.0 Consultation Plan 

The following are the major activities of the NESC consultation: 


Milestone 

Date 

Consultation Initiation 

June 2004 

Team Formation 

August 2004 

Technical Interchange Meeting 

4-6 August, 2004 

Evaluation 

August - September 2004 

Stakeholder Out Briefing 

26 September 2004 

Final Report Submission 

16 May 2005 


The NESC consultation team met with the ET statistics group August 4-6, 2004 at the Michoud 
ET Assembly plant, for briefing and discussion of the dissection data. The NESC team was 
provided with the dissection data as well as already completed analyses of the dissection data. 
Through telecons and face-to-face meetings, the joint statistics team defined the data available, 
specified the assumptions that must be made to establish statistic descriptions of the dissection 
data, and defined limitations on using statistics for certification. Dr. Fayssal Safie presented the 
joint finding to the PRBC on September 30, 2004 (see Appendix B). 


NESC Request No. 04-027-1 



NASA Engineering and Safety Center 
(NESC) Technical Consultation Position 
Paper 


Document #: 

RP-05-57 


Version: 

1.0 


Title: 


Support Provided to the External Tank (ET) Project on 
the Use of Statistical Analysis for ET Certification 


Page #: 

8 of 37 


In addition, the NESC team did independent analyses of the dissection data to establish their 
conclusions. 

4.0 Description of the Problem and Proposed Solutions 

4.1 Problem 

Choosing a distribution to represent the observed data is a critical step in the statistical analysis. 
There currently is no engineering rationale to choose one distribution over another. Several 
distributions may fit the data equally well, and provide very different estimates of the expected 
maximum. The ET program has identified four foam failure modes. Understanding the 
frequency and size of internal defects has become important to understanding and predicting 
debris release. 

The ET has several different types of closed cells foam, various ablator materials, and ice /frost 
that fonn potential contributors to the debris environment around the Space Shuttle on ascent. 

For the return to flight (RTF) certification effort, the major focus has been on the areas of 
manually sprayed foam which were shown to be highly susceptible to having latent defects 
(voids) within the foam that could cause structural failure of the surrounding foam due either to 
entrapped gas pressure or liquid nitrogen /liquid air gasification within the voids on ascent. 

Based on previous flight history and early destructive analysis (dissection), decisions were made 
early in the RTF process to remove manually sprayed foam in the area of the bipod fitting and 
the intertank-to-liquid-hydrogen-tank flange. Since that time, other areas were added to the 
“remove and replace” category based on data acquired in the RTF process. These include the 
forward 10 feet of the Protuberance Airload (PAL) ramp on the liquid hydrogen tank and the 
longeron areas at the aft end of the tank. These “remove and replace” areas are having foam 
reapplied using enhanced processes and more rigorous process controls. The manually applied 
foams include BX-250 and BX-265. BX-265 replaced BX-250 due to environmental issues with 
the blowing agent in BX-250. Generally speaking, BX-250 is installed on tanks through ET-120, 
and BX-265 is installed on existing tanks subsequently processed (ET-121 and forward). BX- 
265 is being applied in all redesign/rework areas. 

Approximately 95 percent of the foam is being left on the tanks as it was originally sprayed with 
no rework or repair. These areas consist mainly of “acreage foam” applied with automated 
processes, but a large number of manually sprayed areas will also be left “as is”. 

In its simplest terms, the certification approach taken by the ET project for the debris 
requirement (cohesive failure mode) is to determine, by test, those combinations of void (defect) 
size /shape and depth within the foam which can cause a divot. This test program is referred to 
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as “critical defect testing.” Defects of “slot” and “cylindrical” shape are tested, based on 
principles of fracture mechanics. 

In order to determine whether an area of foam (referred to as a “closeout”) complies with the 
debris mass requirement, an analysis must be undertaken to determine the process yield of 
defects from that area. If the process yield defect size is less than the critical defect size (i.e., 
regardless of depth within the foam, no critical mass debris will be generated), then the area 
would be considered acceptable. If the process yield exceeds the critical defect size (i.e., at a 
certain range of depths, it would be possible to liberate debris exceeding the allowable limit) then 
further assessments that include local geometry, foam thicknesses, and thermal profiles during 
ascent (e.g., an area that stays cold throughout ascent can have a much larger critical defect size 
without producing a divot than other areas) may be required to determine if the situation is 
acceptable. 

A survey of the technical community would undoubtedly conclude that the most intractable 
issues for certification of the ET for debris lie in the area of “as-built” foams. Knowledge must 
be inferred, either by statistical or engineering methods, from destructive evaluation of one ta nk 
(ET-94), with a few areas from other tanks. 

4.2 Proposed Statistical Methods for ET Certification by ET Program 
Summer 2004. 

The proposed statistical method during Summer 2004 is outlined in Figure 4.2-1 (provided by 
ET program). The 3-sigma estimates of ET foam defect size are all point estimates — distribution 
to be determined. All defects were divided into two types, slots and cylinders, depending on the 
ratio of the largest (LI) and second largest (L2) dimension of the defect. If L1/L2 > 2.5 a defect 
is called a slot, all others are called cylinders. In addition, the defects were labeled by process 
type (elongated cell, rollover, de lamination, gunspit, void). This method assumed that datasets 
are primarily separated by the part that is sprayed (PAL ramp, Liquid Oxygen (LOX) intertank 
flange, bipod, longeron, etc.). 
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Figure 4.2-1. Three-Estimate Approach - Summer 2004 
4.3 New Statistics Approach Proposed in October 2004 

For this analysis two types of defects are defined: 

1. Defects inherent to the process = “Inherent”. (If a defect can be assumed to be in potentially 
any part, numerical treatments will be used to assess possible size and likelihood.) 

2. Defects unique to a particular configuration = “Configuration Dependent”. (Specific defect 
risks are determined by part geometry. 

Each part will be assessed for areas prone to defect generation.) Figure 4.3-1 outlines the new 
approach (provided by the ET program). 
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Figure 4.3-1. New Statistics Approach - October 2004 


Distributions were fitted to the data and 3-Sigma values were derived for the best-fit distribution 
and for the distribution with the biggest 3-Sigma (of distributions considered). These values are 
point estimates that control only the probability that a single defect chosen at random will not 
exceed that size. Frequency of the defects in flight is not considered. 


For each defect estimate for a part, the determination of an expected maximum begins with the 
3 -Sigma from the grouped data. This estimated can be increased by information from part- 
specific geometry or reduced to accommodate any hardware/ foam thickness constraints. 
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5.0 Evaluation of Use of Statistics for Certification Approach 

5.1 Joint Evaluation by ET Program and NESC 

What dissection data is available ? 

The ET program goal is to make inferences about defects on the next ta nk to be flown — to 
support the certification of as-built foam — from the dissection data that has been collected from 
tank 94, and selected areas from tanks 120, 121, and 123. 

Each defect is measured in three dimensions - length, width and thic kn ess. These are sorted by 
largest (LI), 2 nd largest (L2) and smallest (L3). Based on fracture mechanics principles, all 
defects were divided into two types, slots and cylinders, depending on the ratio of the largest 
(LI) and second largest (L2) dimension of the defect. If L1/L2 > 2.5 a defect is called a slot, all 
others are called cylinders. If the defect is categorized as a slot, the CDS of the slot is defined to 
be the smaller dimension, L2; if the defect is a cylinder, the CDS is equal to the largest 
dimension, LI. 

There are several types of void defects - elongated cells, voids, rollovers, delaminations, 
gunspits, etc. Void defects are further classified by the type of foam used - BX or PDL. 
Elongated cells typically are small voids that were judged to be primarily a function of foam 
chemistry, whereas the other defects are primarily a function of the spray process. In order to 
concentrate the analysis on defects due to the spray process, most elongated cells were purged 
from the database by removing defects with Ll< 0.5” and L2 < 0.3” (i.e., defects with any one 
dimension > 0.5” or any two dimensions > 0.3” were kept). Also, void defects from the repair of 
production acceptance test areas were not included. 

The available data were “conveniently”, not randomly sampled, based upon tanks that were 
available for dissection. Since the data were not randomly selected and are available from only a 
few tanks, derived statistics are valid only for the sampled population and not the fleet, unless the 
dissected tanks are found to be typical of the fleet. 

Available dissection data (as of September 2004) 

PALRamps: 94LH 2 , 94L0 2 ,123L0 2 , 120LH 2 (10’), 121LH 2 (10’), 94SRB 

I/T Flanges: 94L0 2 , 94LH 2 , 120LH 2 , 121LH 2 

Flange Thrust Panel 94LH 2 , 120LH 2 

L02 feedline fairing: 94L0 2 , 120LH 2 , 121LH 2 

Feedline Yoke: 94 (incomplete) 

Ice Frost Ramps: 94L0 2 , 128L0 2 , 94I/T, 94LH 2 , 120LH 2 
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Dog Leg C/T cover: 121LEL 
Longeron: 94LEL 
Nosecone: 94 

What are the limitations ? 

The major assumptions that must be made to use statistical analysis of dissection data for 
certification include: 

• The defects found in ET-94 are representative of all defects that are produced during the 
manually-sprayed processes for all in-stock tanks. 

• The distributions of defects, for both slots and cylinders, are homogeneous (stable) and 
the samples are statistically independent. 

The combined team recognized that there are major limitations to generalizing from ET-94 
to other tanks with “use-as-is” foam. These include: 

• Data for some parts are available from only one tank. 

• Data in some cases are a mixture of BX-250 and BX-265 material. 

• Data on some parts have very small sample size (e.g., N=3). 

• Samples are not random — ta nk 94 plus certain parts of other tanks removed in support of 
redesign. 

• Sprayer-to-sprayer differences are not well understood. 

The combined team recognized several major limitations to fitting statistical distributions 
to the dissection data, including: 

• There is no a priori knowledge of the shape of distribution of CDS. 

• There is no engineering rationale to pick a specific distribution. 

• The natural variation of the process is not well understood. 

• There is a lack of random samples (historical) of sufficient size to empirically select a 
distribution. 

• Process controls related to manually-sprayed foam were related to environmental 
parameters and overlap time. Distributions of these process variables have been studied. 
However, the relationship between process control variables and defects is not known. 

• The critical part for predicting the maximum CDS distribution is the upper tail, where we 
have the least information. 
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5.2 Additional Evaluation by NESC Technical Consultation Team 

What is the critical statistic ? 

There are two random variables that can be treated statistically: 

1) The measurement of a single foam defect, selected at random. 

2) The measurement of N foam defects, and then selecting the maximum of the N values. 

These variables are not the same, and in particular the equivalent 3-Sigma point in the 
distribution of (1) is not the same as that of (2). The latter is what is relevant to the basic 
questions posed here. The distribution of the maximum value of a set of random variables can be 
derived from their individual distributions. The single foam defect statistic is an underestimate 
of the maximum expected defect. The single foam defect statistic misleads by underestimating 
the probability of a large defect. See Appendix C for a description. 

What assumptions must be made to perform statistical analysis on the foam dissections 
from ET-94 ? 

A key assumption that must be made is that the defects found in ET-94 are representative of all 
defects that are produced during the manually-sprayed processes for all in-stock tanks. 

Process control was applied to very few variables, and internal defects were not considered a 
process failure when the as-built tanks were produced. Among the variables controlled (and 
documented) are room temperature and humidity, substrate temperature at the beginning of a part 
spray, and component temperatures and proportions. The rise-time between layers and overlap 
time between layers of foam were well-understood by sprayers and quality personnel, though a 
method for timing was not provided and records were not made. Measurements made on witness 
panels and from plug pulls include tensile strength, density, and pull strength. The relationship 
between these control variables and measurements and defect production has not been 
determined. In addition, sprayer to sprayer differences and how they influence defect production 
are not well understood. 

The manually sprayed foam on test tank 94 and as-built tank 120 is predominately BX-250. The 
manually sprayed foam on the rest of the as-built tanks is BX-265. Whether the defects that 
appear in BX-250 and BX-265 are similar in frequency and size is not kn own. The chemical 
formulas for BX-250 and BX-265 vary in not only propellant, but the BX-265 formula was 
modified in hopes that the handling of the foam, in terms of rise-time and overlap time would 
remain about the same. 
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Although this is described as a small change, the data about the effect of this change is only 
understood in terms of tensile strength, compression, and density. Whether the chemical change 
affects any defect production variables (number, size, defect type, defect category, location, 
distribution, maximum characteristic defect) has not been investigated. Since BX-250 is not 
available for controlled testing, this is unanswerable. 

BX-265 is sprayed at a much higher temperature (BX-250 was sprayed at about 1 10 °F, while 
BX-265 is sprayed at about 155 °F). Whether the higher temperature affects defect frequency or 
maximum defect size is unknown. The sprayers were forced to change their arm/hand 
position(s) to avoid hot component feedlines resting on their hands. Whether this awkward body 
position changed defect production is not known. Assuming that defect frequency and size from 
a BX-250 ta nk is representative of a BX-265 ta nk is unjustified. 

What statistical distribution represents the defect dissection data ? 

Traditional statistical methods address the problem of characterizing the distribution of a variable 
from one or more samples. In some cases one has considerable a priori knowledge of the 
distribution. For example, a standard formula with a few parameters may be known to be valid. 
And, often the goal is to predict features of the main core of the distribution — such as its median. 

The problem here is to estimate properties of the distribution of the maximum CDS, but there is 
no useful a priori knowledge about the distribution family and the key properties are features of 
the upper tail of the distribution -so little knowledge is gained per sample about the features. 

Process control variables for manually-sprayed foam were related to environmental parameters, 
component temperatures, and overlap time. Distributions of these process variables have been 
studied. However, knowledge of the process control does not provide the requisite information, 
since the relationship between process control variables and defects is unknown. 

Additionally, witness panels, and plug pulls from sprayed foam were analyzed — density, 
compressions, adhesion, and so forth. Again, the relationship between these foam characteristics 
and defects is unknown. 

For two reasons, fitting of any of the dozens of standard distributions in the statistics literature to 
the dissection data provides no useful information (Note: with small samples, data in the tails of 
the distribution are not expected.): 

1 . A superior fit of one of these distributions to the core of the data provides little or no 

information on whether that distribution provides a more accurate description of the upper 
tail — the critical aspect. Standard goodness-of-fit tests are not sensitive to shape of the tail of 
the distribution. 
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2. Various distributions with quite different tails can provide equally acceptable fits to the 
available datasets. 

Figure 5.2-1 has three fits to one set of data of slots. Each fit is statistically acceptable (Note: the 
estimated 3-Sigma values are as low as .6766 and as high as 1.255). The first might be 
considered too small to cause a significant divot, and the latter could be above engineering 
limits. At this point there is no reason to pick one distribution over another. 


-3.s +3.s 



Figure 5.2-1. Fitting Distributions to a Sample of Dissection Data 
Is the 3-step largest defect approach conservative ? 

In this 3-step approach, the max expected void is defined as the worst of three estimates: 3- 
Sigma estimate for multiple parts/same configuration, 3-Sigma estimate for similar complexities 
or geometries, and maximum observed estimate (See Figure 4.2-1). 

This process does not assure that the estimate will always be wrong in the too large direction — 
despite the claims. Each of the 3-Sigma estimates is contingent on the selection of a distribution 
type by a goodness-of-fit test. The selection of a distribution can be erroneous (a low tail 
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distribution may fit best when a high tail distribution is appropriate). With a choice of 
distribution varying from part to part, a statistical description of this three-step process, such as 
its distribution, is as yet undetermined. 

Is the new largest defect approach an improvement ? 

The same problems experienced with the old approach continue with the new approach (See 
Figure 4.3-1). In addition, partitioning the data into meaningful sets for analysis after the data 
has been collected and examined raises the concern of statistical bias. Keeping the data separate 
by part can be easily justified, but yields very small samples of defects. The present new 
partitioning of the defects into chemistry, process, and geometry or part-specific categories may 
be useful. However, setting the criterion for this division in an unbiased manner is difficult after 
the data has been examined (and does not change the fact that tank-to-tank differences are not 
well understood). 

Although there are some defects that are unique to a part (such as a gap behind a bolt), the 
argument, in many cases, that a particular defect can only happen in an exact location or at a 
specific depth must be viewed with skepticism unless accompanied by solid data and analysis. 


6.0 Findings and Recommendations 

6.1 The Combined Team Finding 

Dissection data statistics have severe limitations and significant uncertainty in predicting the size 

of the largest defect CDS on the next tank. 

6.2 Recommendations to the PRCB by the Combined Team 

1 . Any statistical results should be subject to an engineering evaluation for consistency with 
experience, engineering analysis, and hardware limitations. 

2. Statistical analysis results/estimates should be used in a support role and as information to 
engineering in ET certification. 

3. Data limitations need to be addressed and minimized to improve the credibility of the 
statistical analysis to support RTF decisions (data must address tank-to-tank or part-to-part 
variation). 

4. The ET Program needs to pursue methods beside statistics to predict the largest possible 
defect CDS. 
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6.3 NESC Technical Consultation Team Findings (Three-Estimate 
Approach) 

1 . The largest CDS value in the next sample is the quantity of interest, and should be estimated 
from the distribution of maximum values. Using the distribution of sample values in place of 
the distribution of the maximum of a sample can raise the error probability by orders of 
magnitude. 

2. The assumption that Tank 94 will have the same type, number, and distribution of defects 
sizes as any as-built tanks is unfounded. The assumption that BX-250 and BX-265 foam will 
have the same type, number and distribution of defect size is unfounded. 

3. To predict maximum CDS for the next tank, the critical information needed is the shape of 
the upper tail of the distribution of defect sizes. Adoption of a standard distribution cannot 
be justified, so this shape is unknown and cannot be effectively constrained by the existing 
dissection data. 

4. The Three-Estimate Largest Defect Approach has not been statistically justified nor assured 
to be conservative. 

6.4 NESC Technical Consultation Team Findings (New Statistics 
Approach) 

1 . The largest defect in the next sample is the quantity of interest, and should be estimated from 
the distribution of maximum values. Using the distribution of sample values in place of the 
distribution of the maximum of a sample can raise the error probability by orders of 
magnitude. 

2. The assumption that Tank 94 will have the same type, number, and distribution of defects 
sizes as any as-built tanks is unfounded. The assumption that BX-250 and BX-265 foam will 
have the same type, number and distribution of defect size is unfounded. 

3. To predict maximum defect size for the next tank, the critical information needed is the shape 
of the upper tail of the distribution of defect sizes. Although the amount of data has 
increased by combining all defects in a category, different distributions will still provide very 
different 3 -sigma estimates and choosing a distribution is still tenuous. 

4. The probabilities associated with statistical analyses are only considered defensible if the 
analysis methods have been described prior to the collection of the data. If the analysis has 
been done after the data has been examined, the analysis is considered to be suggestive. New 
data would be required to draw valid conclusions. 
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6.5 NESC Technical Consultation Team Conclusion (Three-Estimate 
Approach) 

Based on the findings above, there is no firm basis for using statistical analysis of the dissected 
data from ET-94 to serve as a statistical distribution model for the certification of as-built foam 
for other tanks. 

6.6 NESC Technical Consultation Team Conclusion (New Statistics 
Approach) 

The “new” approach suffers from the same issues as the “old” approach — wrong statistic (based 
on the distribution of defects instead of the distribution of maximums), unfounded assumptions 
of tank-to-tank and foam-to-foam similarity, and unfounded choice of distribution to represent 
the data. Therefore, there is no basis for using statistical analysis of the dissected data from ET- 
94 to serve as a statistical distribution model for the certification of as-built foam for other tanks. 


7.0 

Lessons Learned 

N/A 


8.0 

References 

N/A 


9.0 

List of Acronyms 

°F 

Degrees Fahrenheit 

CDF 

Cumulative Distribution Functions 

CDS 

Characteristic Defect Size 

ET 

External Tank 

LOX 

Liquid Oxygen 

NESC 

NASA Engineering and Safety Center 

PAL 

Protuberance Airload 

PDF 

Probability Distribution Function 

PRCB 

Program Requirements Change Board 

RTF 

Return to Flight 

SFLC 

Space Flight Leadership Council 

TPS 

Thermal Protection System 
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10.0 Definition of Terms 


Corrective Actions 


Changes to design processes, work instructions, workmanship practices, 
training, inspections, tests, procedures, specifications, drawings, tools, 
equipment, facilities, resources, or material that result in preventing, 
minimizing, or limiting the potential for recurrence of a problem. 


Finding 


A conclusion based on facts established during the assessment/inspection 
by the investigating authority. 


Lessons Learned 


Knowledge or understanding gained by experience. The experience may 
be positive, as in a successful test or mission, or negative, as in a mishap 
or failure. A lesson must be significant in that it has real or assumed 
impact on operations; valid in that it is factually and technically correct; 
and applicable in that it identifies a specific design, process, or decision 
that reduces or limits the potential for failures and mishaps, or reinforces a 
positive result. 


Observation 


A factor, event, or circumstance identified during the 
assessment/inspection that did not contribute to the problem, but if left 
uncorrected has the potential to cause a mishap, injury, or increase the 
severity should a mishap occur. 


Problem 


The subject of the independent technical assessment/inspection. 


Recommendation An action identified by the assessment/inspection team to correct a root 

cause or deficiency identified during the investigation. The 
recommendations may be used by the responsible C/P/P/O in the 
preparation of a corrective action plan. 


Root Cause Along a chain of events leading to a mishap or close call, the first causal 

action or failure to act that could have been controlled systemically either 
by policy/practice/procedure or individual adherence to 
policy/practice/procedure. 
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VOLUME II: APPENDICES 

Appendix A. NESC Request Form PR-003-FM-01 
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NASA Engineering and Safety Center 
Request Form 


Submit this IT A/I Request, with associated artifacts attached, to: nrbexeC.seC'^nasa.gQV . or to 
NRB Executive Secretary, M/S 105, NASA Langley Research Center. Hampton, VA 23681 


Section 1: NESC Review Board (NRB) Executive Secretary Record of Receipt 


Received (mm/dd/yyyy h:mm am/pm) 
3/17/2004 12:00 AM 


Status: New 


Reference #: 04-027-1 


Initiator Name: Randy Galloway 


E-mail: 

thomas.r.galloway@nasa.g 


Center: SSC 


Phone: (228)-688-3281. Ext, 


Mail Stop: HAOQ/NFSC 


Short Title: Statistical analysis support for ET project 


Description: Provide support for ET project descriptive statistical regression analysis being performed to 
characterize current state and predict future state 


Source (e.g. email, phone call, posted on weh): e-mail/phone call 


Type of Request: Support 


Proposed Need Date: 4/16/04 


Date forwarded to Systems Engineering Office (SEP): (mmdd yyyy li:mm am pm): 


Setfion^^j^stemsJ^ngini^ 


Section 2.1 Potential ITA/1 Identification 


Received by SKO: (mm/dd/yyyy h:mm am/pm): 3/18/2004 12:00 AM 


Potential ITA/I candidate? I iVes 1X1 No 


Assigned Initial Evaluator (IE): 


Date assigned ( mm/dd/yyyy): 


Due date for ITA/I Screening (mm/dd/yyyy): 


Section 2.2 Non-ITA/I Action 


Requires additional NESC action (non-IT A/I)? [XlYes T I No 


If yes: 


Description of action: Provide statistical expertise to ET project 


Actionee: Dawn Schaible 


Is follow-up required? QYcs Rl No If yes: Due Date: 


Follow-up status/date: 


If no: 


NESC Director Concurrence (signature): 


Request closure date: 


Section 3: Initial Evaluation 


Received by IE: (mm/dd/yyyy h:mm am/pm): 
Screening complete date: 


Valid ITA/I candidate? [jYes □ No 


Initial Evaluation Report #: NESC-PN- 


Target_Wfl3J?eviewG3atc^ 


NESC Request Form 
NESC-PR-003-FM-01, vl.O 
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Section 4: NRB Review and Disposition of NCE Response Report 

IT A/I Approved: ["TlYes f - ] No Date Approved: Priority: - Select - 

ITA/I I,ead: , Phone ( ) - , x 

Section 5: ITA/I Lead Planning. Conduct, and Reporting 

Plan Development Start Date: 

ITA/I Plan # NESC-PL- 

Plan Approval Date: 

ITA/I Start Date | Planned: | Actual: 

ITA/I Completed Date: 

ITA/I Final Report #: NESC-PN- 

ITA/I Briefing Package #: NESC-PN- 

Follow-up Required? | |Yes | | No 

Section 6: Follow-up 

Date Findings Briefed to Customer: 

Follow-up Accepted: 1 lYes |_| No 

Follow-up Completed Date: 

Follow-up Report #: NESC-RP- 

Section 7: Disposition and Notification 

Notification type: - Select - | Details: 

Date of Notification: 

Final Disposition: - Select - 

Rationale for Disposition: 

Close Out Review Date: 
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Form Approval and Document Revision History 


Approved: 


NESC Director 

Date 


Version 

Description of Revision 

Office of Primary 
Responsibility 

Effective 

Date 

1.0 

Initial Release 

Principal Engineers 
Office 

29 Jan 04 
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Appendix B. Joint Presentation to PRCB 


NESC Request No. 04-027-1 


Statistical Treatment and Limitations for the ET 

As-built TPS Foam 


September 24, 2004 


NESC Request No. 04-027-1 


The information contained in this briefing should be considered “preliminary” and is subject to ITAR control 



• June Space Flight Leadership Council assigned an 
action to NESC and ET project jointly to: 

1) Characterize available dataset 

2) Identify resultant limitations to statistical treatment 
of External Tank as-built foam as part of overall TPS 
certification and 

3) Report to PRCB and SFLC. 
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Statistical Treatment/Analysis 



• Goal 

- Assess the risk that somewhere on the "next" tank 
there will be a defect large enough to divot 


• Statistical treatment 

- Defect data Characterization 

• Descriptive statistics 

• Fitting distribution to data 

- Extrapolation 

• Estimation of the Max void size - upper statistical 
limit on defect size for a specified 

probability/confidence 

• Estimation of risk of as-built foam divoting. 
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A 


TPS foam dissection database 


7 


What is being measured? 

• Foam Defects 

- Defects due to foam chemistry (e.g. elongated cells) 

- Defects due to foam spray process (e.g. voids, rollovers) 

• Focus of the analysis is characteristic defect size (CDS) 

- Defects are measured in three dimensions: LI >L2>L3 

- Slots are defined by LI /L2>2.5, CDS = L2 

- Cylinders are defined by LI /L2<2.5, CDS = LI 

Available dissection data 

PALRamps: 94LH 2 , 94L0 2 ,123L0 2) 120LH 2 (10’), 121 LH 2 (10’), 94SRB 
l/T Flanges: 94L0 2 , 94LH 2 , 120LH 2 , 121 LH 2 


Flange Thrust Panel 94LH 2 , 120LH 2 

L02 feedline fairing: 94L0 2 , 120LH 2 , 121LH 2 

Feedline Yoke: 94 (incomplete) 

Ice Frost Ramps: 94L0 2 , 128L0 2 , 941/T, 
94LH 2 , 120LH 2 


Being updated 


Dog Leg C/T cover: 121LH 2 
Longeron: 94LH 2 
Nosecone: 94 
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Assumptions and Limitations 
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Major Assumptions 

• The defects found in ET-94 are representative of all defects that are 
produced during the manually-sprayed processes for all in-stock tanks. 

• The distributions of defects, for both slots and cylinders, are 
homogeneous (stable) and the samples are independent. 

Major Limitations to Generalization From ET-94 

• Data for some parts are available from only one tank (tank to tank 
variability needs to be expressed in the data) 

• Data in some cases are a mixture of BX-250 and BX-265 material 

• Data on some parts have very small sample size (e.g., N=3). 

• Samples are not random — Tank 94 plus certain parts of other tanks 
removed in support of redesign 

• Sprayer to sprayer differences are not well understood. 
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Assumptions and Limitations (cont.) 
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Major Limitations of Fitting Distribution Shapes To Defect Size 

• It is not unusual to fit distribution shapes to data, under conditions of 
uncertainty. However, the level of uncertainty related to distribution shape is 
high for defect CDS. 

• Distribution related limitations for manually-sprayed foam dissections 

- There is no a priori knowledge of the shape of distribution of defect CDS. 

- There is no engineering rationale to pick a specific distribution. 

- The natural variation of the process is not well understood. 

- There is a lack of random samples (historical) of sufficient size to 
empirically select a distribution 

- Process controls related to manually-sprayed foam were related to 
environmental parameters and overlap time. Distributions of these 
process variables have been studied. However, the relationship between 
process control variables and defects is not known. 

- Process controls for manually-sprayed foam were related to 
environmental parameters and overlap time, but other technique related 
variables were not controlled. 
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Example of Fitting a Standard Distribution 


7 


For predicting the 
maximum CDS, the 
critical part of the 
distribution is the tail, 
where we have the least 
information. 

Standard goodness-of-fit 
tests are not sensitive to 
shape of the tail of the 
distribution. 

For the CDS datasets 
provided, most datasets 
can be fit well by several 
distributions having very 
different tail properties — 
different “3-sigma” 

values. 
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Normal 


Extreme 

Value 


1.255 


Beta 


0.823 


0.6766 


It is common practice to pick the best fit. 
However, picking a distribution, determines 
the “3-sigma” value. 

A distribution that does not statistically fit a set 
of data could be the distribution that best 
represents the shape of the tail 
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1. Dissection data statistics have severe iimitations and significant 
uncertainty in predicting the size of the largest defect CDS on 
the next tank. 

2. Any statistical results should be subject to an engineering 
evaluation for consistency with experience , engineering 
analysis , and hardware limitations 

3. Statistical analysis results/estimates should be used in a 
support role and as information to engineering in ET 
certification 

4. Data limitations need to be addressed and minimized to improve 
the credibility of the statistical analysis to support RTF decisions 
(data must address tank to tank or part to part variation) 

5. The ET Program needs to pursue methods beside 
statistics to predict the largest possible defect CDS. 
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Appendix C. Statistical Methodology 

Consider these two random variables: 

1) The measurement of a single foam defect, selected at random. 

2) The measurement of N foam defects, and the selection of the maximum of the N values. 

If N>1, these variables are not the same, in particular, the equivalent 3-sigma point in the 
distribution of (1) can be much less than that of (2). If one is trying to ensure that none of the 
foam defects exceed a critical size, probability statements about variable (2) are needed to 
characterize the risk. 

Predicting the maximum Characteristic Defect Size for the next tank. 

The largest CDS value in the next sample is the quantity of interest. The distribution of the 
maximum value of a set of random variables can be derived from their individual distributions. 
The result is most easily described in terms of Cumulative Distribution Functions (CDF). If the 
Probability Distribution Function (PDF) of X is P(x), the CDF is defined to be the probability 
that the measured value is < to x: 

CDF(x) = | 0 X P(x') dx’r 

Given a set of N independent variables, say X = { Xi, X 2 , ... X N }, the CDF of their maximum is 
the product of their CDFs: 

CDF( max(X) ) = CDF( Xi ) CDF( X 2 ) ... CDF( X N ) 
so if the X’s are N random “draws” from the same distribution, this equation becomes 

CDF( max(X) ) = CDF( X ) N 


The figure on the following page shows PDFs (derivatives of the CDFs) for the maximum of N 
variables, normally distributed - with zero mean and unit variance. The normal distribution fit to 
the logs of the defect sizes is one of the candidate distributions. Distributions of maximums can 
be derived for any candidate distribution. 
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Distribution of the Maximum of N Normally Distributed Values 



The curve for N=1 is just the normal distribution itself. For increasing Ns, the peak of the curve 
(the most likely value of the maximum of N draws) sifts to larger values as one would expect, 
because, with more tries, the relatively rare values on the high-end of the distribution are more 
likely to be obtained. So, the distribution of the maximum values of a set of samples increases 
with the numbers of measurements in a sample. The variance of distribution of maximums 
decreases with N. 

The equation of interest is: 

P(next max > critical value) = P cr it = 1- CDF( X 

Either we set a P cr i t , for example to a probability equivalent 3-sigma for the normal (1-.9987) 
and solve for a critical value; or we can determine the probability that the next max is greater than 
some critical value — in this case a value set by engineering. 


NESC Request No. 04-027-1 



NASA Engineering and Safety Center 
(NESC) Technical Consultation Position 
Paper 


Document #: 

RP-05-57 


Version: 

1.0 


Title: 


Support Provided to the External Tank (ET) Project on 
the Use of Statistical Analysis for ET Certification 


Page #: 

36 of 37 


The population equivalent 3-Sigma point of a single defect CDS provides an estimate such that 
99.87% of the time a randomly chosen defect from the next tank will be less than this value. 

This estimate does not change with the expected number of defects in a specific part. Using the 
relationship between a single sample distribution and the distribution of maximums, the 
probability that the maximum of the next sample is greater then the single defect CDS equivalent 
3-Sigma point conditional on sample size can be determined. That is, the .13% chance of seeing 
a larger maximum defect could actually be a 12% chance if 100 defects were expected. 



P(observed max > 3 sigma 

Expected N 

single point estimate) 

1 

0.0013 

2 

0.002598 

5 

0.006483 

10 

0.012924 

20 

0.025681 

30 

0.038274 

40 

0.050703 

50 

0.062972 

60 

0.075083 

70 

0.087036 

80 

0.098836 

90 

0.110483 

100 

0.121979 

200 

.33907883 

400 

.40568054 

600 

.54182654 
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